Skip to main content

Managing and Monitoring Qarbine Hosts

Overview​

You can review log messages in the following manners:

  • SSH and
  • Administration Tool.

For SSH, once connected navigate to ˜/.pm2/log. In that folder you will find various log files.

Administration Tool Log File Reviewing​

In the Administration Tool activate the “Log Files” tab.

  

Click

  

Choose a Qarbine compute node. Multiple Qarbine endpoint configurations are not available in the Professional edition. Also, multiple endpoints on the same host share a logging file.

  

Click

  

The left hand file listing area is populated as shown below.

  

The middle area toolbar includes file navigation buttons    for first, previous, next, and last chunks respectively.

Click   to download the log file.

File position feedback or setting can be adjusted using the entry field

  

The amount of text shown at a time can be adjusted using the entry field

  

Refresh the content by clicking on   .

See the Administration Tool help folder named “Utilities” for information on reviewing user interaction events.

This area of the Administration Tool accesses files on the Qarbine compute nodes. If you SSH into the host, then review the files in the /home/qarbine/.pm2/logs folder. The “serviceName-out.log” (i.e., “main-out.log”) file is a likely source of useful information.The Apache logs are in the /etc/logs/https folder.

Overview​

In most cases you can use the Qarbine Administration tool to manage the Qarbine processes across your Qarbine compute nodes. In some cases you may need to manage the processes in a more direct manner by using a shell interacting with the compute node’s host.

SSH into the instance.

Navigate to the folder

cd ~/qarbine.service

By default the Qarbine services are configured to start upon reboot. This was setup by running

sudo systemctl start qarbine

You can then use standard Linux commands to review the host operations.

Log Files​

There are Apache and Qarbine application log files. The Apache files are located at /var/log/httpd. The application log files are located in ˜/.pm2/log. They can be reviewed interactively as described in the section above. Qarbine uses PM2’s log rotate plugin. More information can be found at https://pm2.keymetrics.io/docs/usage/log-management/.

PM2 Usage​

The Qarbine endpoint service processes are usually started and stopped using the node.js pm2 utility. For utility summary information see https://pm2.keymetrics.io/docs/usage/quick-start/.

In standard operations Qarbine writes out a pm2 configuration file using the name of the Qarbine service as a prefix. The file name format is serviceName.config.js. Qarbine then requests pm2 to start the service as defined within that configuration file using

pm2 start serviceName.config.js

The pm2 utility oversees the Qarbine node.js process execution. It will restart the node.js process if it terminates because of an exception. The delay for restarting is set at 30 seconds. This delay value is defined in pm2ServiceTemplate.txt as

watch_delay: 30000,

Be aware that a new serviceName.config.js file is written out each time Qarbine starts the service.

Qarbine will stop a specific service using the request

pm2 stop serviceName.config.js

All of the pm2 started services may be stopped using

pm2 stop all

To list the running services run

pm2 list

To bounds all of the pm2 managed services run

pm2 restart

The log files are contained with ˜/.pm2/logs. They may also be viewed using the Qarbine Administrator tool’s “Log Files” tab.

Note that Qarbine does not request pm2 to persist the process information about all of the running services. This is because the Qarbine Administration tool utility page is used to manipulate which Qarbine services are enabled, disabled, and active.

PM2 logrotate config has some of the default values

  • max_size: By default, it is 10 MB. If the log file size is 10 MB, then it will run the pm2-logrotate and will make a new log file.
  • retain: By default, this value is 30, which means after 31 files are generated,the old log file will be deleted.
  • compress: By default, it is false.
  • dateFormat: YYYY-MM-DD_HH-mm-ss this is by default.
  • rotateInterval: By default will set 0 0 * * * everyday at midnight.

Utility Commands​

Several utility commands are available within ˜/qarbine.service

  • dumpBuildInfo.sh
  • dumpCertificateSubject.sh
  • dumpCertificateInfo.sh
  • dumpLocation.sh
  • startApache.sh
  • stopApache.sh
  • whatIsRunningOnPort.sh
  • whatIsPidCommandLine.sh
  • whatsUp.sh

Monitoring Application Health​

Overview​

See the Monitoring and Managing Services document in the Tools;Administrator section of the online documentation for reviewing application operations using the Qarbine Administration tool.

AWS Deployment Health Checking​

The following steps can be used to view your instance status from an AWS perspective.

  1. Navigate to your Amazon EC2 web console
  1. Verify that you're in the correct region
  1. Choose the Instance tab
  1. Select your launched instance by checking the checkbox   
  1. Review the instance’s metadata page content
  1. Choose the Status checks tab at the bottom of the page to review if your status checks passed or failed.

  

Azure Deployment Health Checking​

See this link for information on Resource Health overview on options to monitor your VM health.
https://learn.microsoft.com/en-us/azure/service-health/resource-health-overview

See this link for suggestions on using Azure Metrics Explorer and metric alerts
https://learn.microsoft.com/en-us/training/modules/monitor-azure-vm-using-diagnostic-data/

General Usage Instructions​

See the Deployment Considerations document in the Gettings Start;Deployment online documentation section for more information regarding sensitive information,encryption, and instance health monitoring.

3rd Party Enterprise Monitoring Tools​

This feature is dependent on the Enterprise Edition of Qarbine. When configured, cloud monitoring agents such as Datadog, Grafana and NewRelic provide monitoring features for host, Apache, internal MongoDB database, and node,js metrics. Such monitoring is not part of the Professional nor Team edition. The Qarbine integration is described in their respective documents.